Method and apparatus for a head display unit with a movable high resolution field of view

ABSTRACT

This patent provides a method and apparatus for improving the spatial resolution of the display, so that it appears less pixelated. This is accomplished by generating a first portion of an image that corresponds to a user&#39;s fovea and a second portion of the image that corresponds to areas of non-fovea regions of the retina. For the region of the image that corresponds to the fovea, the display uses high spatial resolution of pixels, so that the display is optimized for the user&#39;s central vision which has the highest visual acuity. For regions of the image that do not correspond to the fovea, the display uses a lower spatial resolution of pixels. Furthermore, the first portion of the image with the high spatial resolution can be moved within the image so as to accommodate various look angles and convergence points of the user.

TECHNICAL FIELD

Aspects of this disclosure are generally related to display of images onextended reality headsets.

BACKGROUND

Three-dimensional medical images can be presented via an augmentedreality, virtual reality or mixed reality headset. Key strategies forpresenting imaging on head display units include those disclosed in U.S.Pat. Nos. 8,384,771, 9,349,183, 9,473,766, and 9,980,691.

SUMMARY

All examples, aspects and features mentioned in this document can becombined in any technically possible way.

In some implementations, a user controller interface for presentation of3D medical images comprises a joystick and functional buttons.Functionality provided by the buttons for review digitally may include,but is not limited to: a) changing orientation (roll, pitch, yaw) of a3D cursor; b) zooming viewpoint toward and away from the 3D cursor; c)invoking convergence; d) raising and lowering where the 3D cursor isdisplayed on the headset; e) changing the size, shape, and color of the3D cursor; e) invoking filtering, segmentation, sequencing, statistical,and reporting operations; f) invoking pointer and movement controlthereof; g) annotating one or more 3D cursors within the volume ofinterest; and h) invoking icon options.

Some implementations include tangible equipment items with position andorientation tracking, possibly including, but not limited to, one ormore of: a desk registration component equipped with registrationpoints; a focal point pen with registration points, position andorientation tracking; a platform with registration points, position andorientation tracking; a multifunction tool with registration points,position and orientation tracking; a head display unit (e.g., augmentedreality, virtual reality or mixed reality) equipped with registrationpoints, position and orientation tracking and innovative convergencecapabilities; a cutting device equipped with registration points,position and orientation tracking; and, a catheter device, which couldalso have the option for tracking. The items may enable performance ofvirtual tasks possibly including: control of a digital pointer; controlof a 3D cursor; interaction with virtual tissues through the hand-heldtools, such as pulling virtual tissues together or spreading them apart;step-through sequencing; and annotation. Implementation of these virtualtasks may include filtering, segmentation, voxel manipulations andcoordinated multi-voxel shifts, as disclosed in U.S. patent applicationSer. No. 15/904,092 titled PROCESSING 3D MEDICAL IMAGES TO ENHANCEVISUALIZATION and U.S. patent application Ser. No. 16/195,251 titledINTERACTIVE VOXEL MANIPULATION IN VOLUMETRIC MEDICAL IMAGING FOR VIRTUALMOTION, DEFORMABLE TISSUE, AND VIRTUAL RADIOLOGICAL DISSECTION, both ofwhich are incorporated by reference.

Some implementations comprise a display icon that illustrates a humanbody facing in a direction specified by the medical person viewing themedical images. The icon may comprise one or more of: a) a depiction ofthe current viewing point properly located relative to the human bodyicon; b) a depiction of the outline of volume of interest being examinedwithin the human body icon; and c) the location of the 3D cursor.Further, during the course of the examination wherein the 3D cursor andcontents would have been extracted-manipulated to some degree (e.g.,voxel changed from initial orientation rolling, pitching and/or yawcommands to a new orientation) , the medical personnel could specify anew icon that depicts to 3D cursor along with both initial viewing pointand current viewing point properly located with, for example, an arroworiginating at the initial viewing point and terminating at the currentviewing point.

Some implementations comprise one or more of: a geo-registrationcoordinate system; geo-registered volumetric medical images; headdisplay unit with tracking and orientation; focal point pen withtracking and orientation; a virtual pedestal/platform with tracking andorientation; a knife with tracking and orientation, a 3D cursor withtracking and orientation, and a desk registration component. Using thetechnique described in U.S. patent application Ser. No. 15/949,202, thepatient's medical images would be associated with a volumetriccoordinate system. A focal point pen could be registered with thevolumetric coordinate system. Using a geo-registration point on the headdisplay unit, the head display unit could be geo-registered with thevolumetric coordinate system. Further, using a geo-registration point ona knife unit, the knife unit could be geo-registered with the volumetriccoordinate system. A virtual pedestal/platform could be registered withthe volumetric coordinate system. Further, a 3D cursor (e.g. asdescribed in U.S. Pat. No. 9,980,691) could be geo-registered andmoveable via commands entered by the medical personnel using a handcontrol unit. A copy of the 3D cursor contents could be made and sent toother locations for examination (e.g., place on pedestal/platform).Thus, geo-registration of various components with the patient's medicalimages is enabled in an overall volumetric coordinate system.

Some implementations include a virtual environment comprising at leastone of: a geo-registration coordinate system; a head display unit withtracking and orientation; a focal point pen with tracking andorientation; a multi-function tool tracking and orientation; a virtualpedestal/platform with tracking and orientation; and a desk registrationcomponent. The patient's medical images would be associated with orinclude a volumetric coordinate system as described above. The focalpoint pen, head display unit, desk registration component, and virtualpedestal/platform could each be independently registered with themedical image volumetric coordinate system.

Some implementations include head movement and head orientationtracking. Orientation and movement may be measured by an inertialmeasurement system within the head display unit. Further, the headdisplay unit would have a registration point which would, when touchingthe geo-registration component, in conjunction with geo-processingsoftware, enable geo-registration of the head display unit within themedical image volumetric coordinate system. Movements of the headdisplay unit in location (i.e., X, Y, and Z coordinates) and orientation(i.e., roll, pitch, and yaw) would be transmitted to a computer via atransmitter within the head display unit. The computer would compute howthese head movements would affect the 3D volume being displayed andcreate an adjusted volume for display. The adjusted volume would betransmitted by the computer and received by the receiver element in thehead set and subsequently shown on the head display unit eye pieces.

The geo-registered focal point pen may be a geo-registered physicalobject to be held in the hand of medical personnel. The focal point penunit would have a registration point which would, when touching thegeo-registration component, in conjunction with geo-processing software,enable geo-registration of the focal point pen unit within the overallsystem containing the volumetric medical images and other systemcomponents. These movements of the focal point pen in location (i.e., X,Y, and Z coordinates) and orientation (i.e., roll, pitch, and yaw) wouldbe obtained by an inertial measurement unit within the pen andtransmitted to the computer via a transmitter also within the pen.Functionality of the focal point pen could comprise one or more of thefollowing: a) moving the focal point pen within the 3D image set so asto follow arteries/veins within a complex vascular structure; b)touching a point within the 3D image set with the tip of the pen forannotation and/or cross reference the a particular 2D image slice; c)writing notes, drawing symbols (e.g., encircle tissue of concern; drawarrows) and; d) color coding could be invoked by medical personnel.

Some implementations include a method of using the focal point pencomprising one or more of the following steps: a) moving the focal pointpen within the 3D image set so as to follow arteries/veins within acomplex vascular structure; b) touching a point within the 3D image setwith the tip of the pen for annotation and/or cross reference the aparticular 2D image slice; c) writing notes, drawing symbols (e.g.,encircle tissue of concern; draw arrows); d) selecting tissue type toassign physical property (e.g., assigning bone a rigid physicalproperty); and e) selecting tissue types for where to add voxels so toseparate tissues apart to better visualize complex anatomical structures(e.g., separating the tangle of blood vessels in a cerebralarteriovenous malformation).

Some implementations include a method for assigning physical propertiesto tissue types, comprising one or more of the following steps: a)assigning a stretchable property to voxels of certain tissue types(e.g., muscle, tendon, etc.); b) assigning points of fixation (e.g.,tendon or ligament insertion site into the bone); c) assigning a rigidproperty to voxels of certain tissue types (e.g. bone); d) assigning arigid, but mobile property to tissue type (e.g., shoulder's ball andsocket joint); and e) assigning non-fixed, fluid property to certaintissue types (e.g., blood inside blood vessel).

Some implementations include calibration to ensure component accuracy.To verify the accuracy of the geo-registered components, it may bedesirable to perform a check using calibration points. The first step isto have the medical personnel arbitrarily place 6 or more calibrationpoints into the geo-registration coordinate system. The next step is forthe medical personnel to touch each of the points with the systemcomponents and check that the coordinates shown by the component matchthose of the calibration points.

The geo-registered pedestal/platform may be a geo-registered physicalobject to be held in the hand of medical personnel. Thepedestal/platform unit may have a registration point which would, whentouching the geo-registration component, in conjunction withgeo-processing software, enable geo-registration of thepedestal/platform unit within the overall system containing thevolumetric medical images and other system components. These movementsof the pedestal/platform in location (i.e., X, Y, and Z coordinates) andorientation (i.e., roll, pitch, and yaw) would be obtained by aninertial measurement unit within the pedestal/platform and transmittedto the computer via the transmitter also within the pedestal/platformunit. Functionality of the pedestal/platform may include moving thepedestal/platform to a volume of interest within the 3D medical image(e.g., volume contained within the 3D cursor) by hand movements of thepedestal/platform and then the medical person viewing the medical imagesissues command to affix the volume of interest to the pedestal/platform.The medical person viewing the medical images by hand control of thepedestal/platform could rotate, tilt the pedestal/platform and move thepedestal/platform closer/further for examination. Additionally, thisexamination process could be accompanied by head movements by medicalperson viewing the medical images so as to obtain a better perspectiveof any tissue of potential concern.

Some implementations include moving the pedestal/platform to a volume ofinterest within the 3D medical image (e.g., volume contained within the3D cursor) by hand movements of the pedestal/platform and then themedical person viewing the medical images issues a command to affix thevolume of interest to the pedestal/platform. The medical person viewingthe medical images by hand control of the pedestal/platform couldrotate, tilt, or translate the pedestal/platform for examination.Further, this examination process could be accompanied by simultaneoushead movements by medical person viewing the medical images so as toobtain a better perspective of any tissue of potential concern.

Some implementations include a geo-registered 3D virtual cursor. The 3Dvirtual cursor might have some of the features described in U.S. Pat.No. 9,980,691 and U.S. patent application Ser. No. 15/878,463, both ofwhich are incorporated by reference. The contents within the 3D virtualcursor could be copied and moved within the overall geo-registrationsystem to a different geo-registered position (e.g., thepedestal/platform). The 3D virtual cursor contents could be affixed tothe pedestal/platform and moved in concert with the pedestal/platformmovements. The 3D virtual cursor movements and selection of contentswould be at the at the command of medical personnel through the controlunit to the computer.

Some implementations include an ablation method. The ablation techniquecould be used in conjunction with a 3D digital mass, transported by the3D cursor as described above and affixed to geo-locatedpedestal/platform. The first step may be determining the outer ‘shell’of an organ of interest to the medical person viewing the medical images(e.g., using segmentation techniques described in U.S. patentapplication Ser. No. 15/904,092, which is incorporated by reference);sequentially eliminating one voxel deep layers from the outer surface atthe direction of the medical person viewing the medical images;alternatively or additionally, selecting one layer in the X, Y, Zcoordinate system (e.g., select the X-Y layer with the highest Zcoordinate and eliminating that layer at the direction of the medicalperson viewing the medical images.

The geo-registered ‘knife’ may be a geo-registered physical object to beheld in the hand of medical personnel. The knife unit may have aregistration point which would, when touching the geo-registrationcomponent, in conjunction with geo-processing software, enablegeo-registration of the knife unit within the overall system containingthe volumetric medical images and other system components. Thesemovements of the knife in location (i.e., X, Y, and Z coordinates) andorientation (i.e., roll, pitch, and yaw) would be obtained by aninertial measurement unit within the knife and transmitted to thecomputer via a transmitter also within the knife. The geo-registeredknife could be used by medical personnel to ‘carve away tissue’ from ageo-registered 3D digital mass. The geo-registered knife could be usedin conjunction with a 3D digital mass mounted on the virtualpedestal/platform and/or within the 3D cursor. The geo-registered knifewould be geo-registered within the geo-registration system and could beused by the medical person viewing the medical images to pick up thegeo-registered knife and move it to the 3D digital mass of currentinterest, then pass the geo-registered knife through the 3Dgeo-registered digital mass. Tissue which is external to the surfacecreated by the geo-registered knife when it passed through the 3Dgeo-registered digital mass (the side of the geo-registered knifepre-selected by the medical person viewing the medical images for 3Dtissue extraction form the 3D digital mass) would be virtually removed.The knife may include an exact registration point (e.g., tip ofgeo-registered knife), additional geo-registration points to indicatethe cutting surface of the knife and an internal measurement unit toprovide changes in the X, Y, Z coordinate system and also roll, pitchand yaw of the knife.

Some implementations include a multi-function tool that can be used forgrabbing a tissue type within the 3D medical volume. Steps may includemoving the multi-function tool to a volume of interest within the 3Dmedical image (e.g., volume contained within the 3D cursor) by handmovements of the multi-function tool and then the medical person viewingthe medical images issuing a command to affix a portion of the tissuesubtype to the multi-function tool. The medical person viewing themedical images by hand control of the multi-function tool could affixthe multi-function tool to a tissue and move the selected tissue (i.e.exert force on the tissue structure to translate, rotate, pull it) suchthat the selected tissue moves in accordance with its assigned physicalproperty and the assigned physical properties of the nearby adjacentstructures.

In some implementations the multi-function tool may be used for cuttinga tissue type within the 3D medical volume. Steps may include moving theMFT to a volume of interest within the 3D medical image (e.g., volumecontained within the 3D cursor) by hand movements of the MFT and thenthe medical person viewing the medical images issuing a command to cut aportion of the tissue subtype using the multi-function tool. The medicalperson viewing the medical images by hand control of the multi-functiontool could affix the multi-function tool to a tissue and move theselected tissue (i.e. exert force on the tissue structure to translate,rotate, pull it) such that the selected tissue moves in accordance withits assigned physical property and the assigned physical properties ofthe nearby adjacent structures.

Some implementations include utilizing the multi-function tool forfixing a tissue type within the 3D medical volume. Steps may includemoving the multi-function tool to a volume of interest within the 3Dmedical image (e.g., volume contained within the 3D cursor) by handmovements of the multi-function tool and then the medical person viewingthe medical images issuing a command to attach one tissue subtype toanother tissue subtype using the multi-function tool.

Some implementations include a 3D geo-registered catheter. The 3Dgeo-registered catheter may be used in conjunction geo-registration ofmedical images as described in U.S. patent application Ser. No.15/949,202 or U.S. Pat. No. 9,301,809, both of which are incorporated byreference. The radiologist/interventionist could switch back and forthbetween the geo-registered 3D system using the 3D head mounted displayand standard displays currently available in interventional operations.This permits utilization of distance markers and 3D screen capturesgenerated during pre-operative planning. Further, alerts could be givenin near real time as critical junctions were being approached.

In some implementations the geo-registered catheter is used inconjunction with a 3D digital image of the vascular structure within thepatient. The catheter could continuously compute the total distancetravelled which could be displayed, and time-tagged and recorded forlater review. The geo-registered catheter could be used duringpre-operative planning of an interventional procedure including, but notlimited to, treatment of a middle cerebral artery aneurysm. Steps mayinclude: inserting the geo-registered catheter into the 3D digitalvascular structure at a pre-determined point such as the groin of thepatient into the common femoral artery, then the external iliac artery,then the common iliac artery, then the abdominal aorta, then thethoracic aorta, then the brachiocephalic artery, then the common carotidartery, then the internal carotid artery then the middle cerebralartery, and finally into the aneurysm. Augmented reality distancemarkers could be added to each intersection. Each succeeding element ofthe catheter goes to the location and orientation of the immediatelyproceeding (or trailing) element as the radiologist pushes (or pulls) onthe catheter.

The user controller interface may enable medical personnel reviewing 3Dmedical images to review inputs provided via geo-registered tools.Functionality for review virtually may include, but is not limited to,the following when interfacing with the 3D cursor: a) changing theorientation of the 3D cursor roll, pitch and yaw, such as moving thehand-held geo-registered platform temporarily attached to the 3D cursorof interest; b) zooming the medical person viewpoint in toward the 3Dcursor and out away from the cursor, such as moving the hand-heldplatform temporarily attached to the 3D cursor closer to the left andthe right eye viewing perspectives or a leaning forward action to movethe persons eyes closer to their hand; c) invoking convergence, such asmoving a focal point pen acting as a convergence point for objects inthe volumetric medical imaging database; d) raising and lowering the 3Dcursor as to where it is displayed on the headset, such as raising orlowering the radiologist's chair and keeping the radiologist's desk,master control platform, and all other geo-registered tools in a fixedposition; e) changing the size, shape, and color of the 3D cursor, suchas using the multi-function tool to pinch the 3D cursor to make itsmaller; e) invoking filtering, segmentation, sequencing, statistical,and reporting operations, which can be performed via a variety ofgeo-registered tools; f) invoking pointer and movement control thereof,such as affixing the virtual pointer to the focal point pen so that asthe radiologist moves the geo-registered 3D pen that he/she is holdingin his/her hand, so does the virtual pointer moves through space in thevirtual images projected on the head display unit; g) performing virtualcutting by moving the geo-registered cutting tool, which is linked to avirtual scalpel; h) performing virtual movement of a geo-registeredcatheter which is linked to a virtual catheter.

Any combination of the components and steps could be combined. Examplesmay include: a) the medical personnel moving the geo-registered 3Dcursor within the geo-registered 3D volumetric data to a volume ofinterest; b) copying and capturing data within the geo-registered 3Dcursor; c) transporting the captured volume to and affixing this data tothe pedestal/platform; d) medical personnel picking up thepedestal/platform (i.e., a geo-registered physical object to be held inthe hand of medical personnel), turning, tilting, rotating, bringingcloser/further away during examination of contents; e) picking up thefocal point pen platform (i.e., a geo-registered physical object to beheld in the hand of medical personnel) with the other hand, pointing toan element of the transported 3D contents, and drawing an arrow to thiselement; f) sending contents to report file; g) laying down the focalpoint pen and pick up knife; h) using the knife, dissecting the contentson the platform; and i) sending the dissected contents to the reportfile.

Some implementations comprise converging the left and right eyes to asingle convergence/focal point through adjusting the display to accountfor extra-ocular muscle movements to change the look angle of the eyeand accommodation to change the shape of the lens/pupil of the eye as itrelates to the field of view (See U.S. Pat. No. 9,349,183, which isincorporated by reference). Process steps may comprise one or more ofthe following: a) occluding/eliminating the left portion of the displayand associated voxels which would have been displayed with additionalvoxels displayed on the right portion of the display, if lookingstraight ahead, and similarly for the right eye occluding/eliminatingthe right portion of the display; b) shifting the display of theconvergence/focal point to the right of the center point of the displayproportional to the angular change based on distance of the view pointfrom the convergence/focal point, similarly shifting theconvergence/focal point to the left for the right eye; and c) reducingthe total voxels displayed for both eyes to reflect changes in the fieldof view when observing a close object (i.e., accommodation).

The pixel display of the HDU may be variable in terms of the angularresolution per pixel. This altering angular field of view could changein a step-wise fashion or in a non-linear fashion. Accompanying changesin brightness could be used to highlight the region of differentresolution bands. Nominally, the high-resolution band would beassociated with the fovea region. The location of the high-resolutionband could vary based on the location of the focal point pen within theoverall display.

In some implementations, fixed focal point spots can be placed withinthe volumetric images to help the radiologist focus on criticalstructures. In switching from one focal point spot to the next, theradiologist's eyes can jump from spot to spot via saccadic eyemovements.

In some implementations, a mobile focal point spot can be placed withinthe volumetric images to help the radiologist focus on criticalstructures in a more comprehensive manner than jumping from one spot tothe next. The movements of the focal point spot can be directed by anartificial intelligence algorithm or via radiologist-directed control ofthe virtual pointer or the focal point pen. Note that the virtualpointer is a virtual object whereas the focal point pen is a tangibleobject. Both of these objects operate in virtual space.

In some implementations, an electronic database of known pathology iscalled upon for comparison with an unknown pathology lesion from thepatient's current scan. A known pathology dataset could be built andcalled upon. The volumetric dataset is used to generate a 3D image in avirtual cursor of the pathology affixed to a second geo-registeredplatform. Multiple different virtual pathologies can be placed onmultiple individual geo-registered platforms for teaching purposes aswell as examination purposes.

In some implementations, software to geo-register the above componentsand operate the system during an examination is included.

BRIEF DESCRIPTION OF THE FIGURES

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 illustrates presentation of a three-dimensional (3D) medicalimage by a HDU (head display unit).

FIG. 2 illustrates aspects of manipulation of three-dimensional medicalimages with true stereoscopic viewing.

FIG. 3 illustrates a radiologist's work station that includes aplurality of geo-registered tools for manipulating three-dimensionalmedical images.

FIG. 4 is a flow diagram of steps for using geo-registered tools tomanipulate three-dimensional medical images for examinations.

FIG. 5 illustrates the master control platform in greater detail.

FIG. 6 illustrates the HDU in greater detail.

FIG. 7 illustrates the geo-registered focal point pen within thegeo-registered coordinate system.

FIG. 8 illustrates multiple calibration points within thegeo-registration coordinate system.

FIG. 9 illustrates the geo-registered hand-held pedestal within thegeo-registration coordinate system.

FIGS. 10A, 10B, 10C and 10D illustrate an ablative process to aid insearching the internal structure and any abnormalities of an organ.

FIG. 11 illustrates a geo-registered knife used to dissect tissueswithin the geo-registration coordinate system.

FIG. 12 illustrates a geo-registered multi-function tool used tomanipulate voxels within the geo-registration coordinate system.

FIG. 13 illustrates the geo-registered catheter with navigation througha blood vessel.

FIG. 14 illustrates the geo-registered tool inputs to provide examplesof true stereoscopic 3D viewing techniques in the “virtual world”.

FIGS. 15A and 15B illustrate digital convergence of the head displayunit to a focal point.

FIGS. 16A and 16B illustrate a process of providing increased resolutionin the field of view corresponding to the fovea and is linked to thefocal point.

FIGS. 17A and 17B illustrate a visual representation of convergence.

FIGS. 18A, 18B, 18C, and 18D FIG. 18 illustrate a method of helping theradiologist in search pattern via utilization of saccades searchtechnique.

FIGS. 19A, 19B, 19C, and 19D illustrate a method of helping theradiologist in search pattern via utilization of smooth tracking searchtechnique.

FIGS. 20A and 20B illustrate the capability of using the geo-registeredplatform to display a known pathology from a database next to anothergeo-registered platform with an unknown pathology from the patient'sscan.

DETAILED DESCRIPTION

Some aspects, features and implementations described herein may includemachines such as computers, electronic components, radiologicalcomponents, optical components, and processes such ascomputer-implemented steps. It will be apparent to those of ordinaryskill in the art that the computer-implemented steps may be stored ascomputer-executable instructions on a non-transitory computer-readablemedium. Furthermore, it will be understood by those of ordinary skill inthe art that the computer-executable instructions may be executed on avariety of tangible processor devices. For ease of exposition, not everystep, device or component that may be part of a computer or data storagesystem is described herein. Those of ordinary skill in the art willrecognize such steps, devices and components in view of the teachings ofthe present disclosure and the knowledge generally available to those ofordinary skill in the art. The corresponding machines and processes aretherefore enabled and within the scope of the disclosure.

FIG. 1 illustrates a virtual image displayed in a HDU (head displayunit) 108. The radiologist's workstation will include a desk 100, acomputer system 102, a diagnostic monitor 104, a controller 106 and ahead display unit (HDU) 108. The HDU 108 can display a 3D cursor 110which contains a sub-volume of interest 112. In this illustration, thediagnostic monitor is displaying a figure of a person with the 3D cursor110 located over the right upper quadrant of the abdomen. The virtualimage presented by the head display unit 108 includes a representationof the 3D cursor 110 and tissues within the volume of the 3D cursor,e.g. a liver. In a current “digital world” scenario the head displayunit lacks position and orientation tracking. To manipulate the images,options beyond the standard keyboard and mouse controls may include theuse of a controller 106 and joystick 106. In a “virtual world” scenariothat will be described below the head display unit has a position andorientation tracking feature so movement of the head can be used tomanipulate the virtual image. Further, geo-registered tools may also beused for manipulation of the virtual image.

FIG. 2 illustrates controller/joystick inputs to provide examples oftrue stereoscopic 3D viewing techniques in the “digital world”. In thisfigure, representative examples of viewing options available through theuse of a hand-held controller equipped with joystick (not shown) areillustrated. In the center of the figure, the HDU 200 is shown with aninitial left eye view point 202, left eye viewing angle 204 and a volumeof interest 206 (e.g., volume-subtending 3D cursor) as well as a righteye view point 208, right eye viewing angle 210 and the volume ofinterest 206 (e.g., volume-subtending 3D cursor). Note that the HDU 200displays the volume of interest 206 with a left eye image 212corresponding to the left eye view point 202 and left eye viewing angle204 of the volume of interest 206 and a right eye image 214corresponding to the right eye view point 208 and right eye viewingangle 210 of the volume of interest 206. In the first example,controller/joystick (not shown) input can direct the 3D cursor to changeorientation (roll, pitch and yaw) 216. Next, controller/joystick inputcan alter the distance between the left eye view point 218 and right eyeviewpoint 220 with respect to the volume of interest 222. The imageillustrated shows the left eye view point 218 and right eye view point220 moved closer (i.e., zoomed in) towards the volume of interest 222.The volume of interest 222 could alternatively be moved in itscoordinates toward the left eye view point 218 and right eye view point220 to achieve the same zoomed in effect. Next, the controller/joystick(not shown) input can direct convergence to a focal point 224 shown asthe orange circle within the center of the 3D cursor 226. Note the lefteye viewing point 228 and the right eye viewing point 230. Also note theleft eye viewing angle 232 has been adjusted based on the convergence tothe focal point 224. Also note that the right eye viewing angle 234 hasalso been adjusted based on the convergence to the focal point 224.Further, note that a line 236 can be extending from (or close to) themidpoint of the HDU 238 to (or close to) the convergence point 224. Wewill refer to this line 236 as the center line of focus. Note that itwould be possible to help the user focus on particular structuresthrough implementation of the center line of focus 236. Next,controller/joystick (not shown) input can direct raising or lowering ofthe 3D cursor 242 within the HDU 200, or moving the 3D cursor from sideto side (not shown). Next, controller/joystick (not shown) input canchange the size 244, shape 246 or color 248 of the 3D cursor. Next,controller/joystick input can invoke filtering 250, segmentation 252,sequencing 254, statistical analysis 256 and reporting 258, which werediscussed in USPTO application Ser. No. 15/904,092. Next,controller/joystick input can direct movement of a virtual pointer 260through the volume of interest 262 from an initial position 264 to asubsequent position 266. The pointer 260 operates within the 3D volume262. Movement of the pointer 260 would be controlled by person viewingthe images. The pointer could vary in appearance to take on any form ofone, two- or three-dimensional objects and could be programmed to moveautomatically without the user control. The pointer 260 is useful whensmooth pursuit eye movements are necessary. For example, smooth pursuiteye movements would be beneficial when examining arteries for anyblockages, wherein using the pointer 260 to trace along the length ofarteries looking for blockages. Saccadian eye movement could resultskipping over portions of the artery and a serious blockage goundetected; therefore, the pointer 260 could be helpful in aiding thissearch pattern. Multiple colored/shaped pointers 260 could be used totrace the different flows of arteries and veins. Next,controller/joystick (not shown) input can direct annotations (not shown)of one or more 3D cursors 268 within the volume of interest 262.Finally, controller/joystick input can direct icon options 270 asrelated to the volumetric medical imaging 262 to keep the radiologistorganize in his/her checklist approach to a complex exam. Note the humanbody icon 270 to the display. During the course of the examination ofthe volume by the medical person viewing the medical images, it may beuseful to quickly refer to an icon 270 in order to re-orient whereexactly in the body is some tissue of interest/concern. The icon 270would also be of utility in discussions between medical personnel. Thisfigure shows the body in a vertical position facing forward. This iconshows: a) the current or initial viewing point relative to the humanbody; b) the outline of the total volume (or sub-volume) being examined;and c) approximate location of the 3D cursor(s) within the human bodyicon. Orientation of the body would be under the control of the medicalperson viewing the medical images, as would whether to display the icon270 or not.

FIG. 3 illustrates a top down view of the radiologist's work station. Inaddition to standard items present at radiology work stations includinga computer 300, keyboard 302, mouse 304, voice recorder 306 and monitors308, multiple additional components are present in this patent. First,is the master control platform 310, which has registration point(s) 312and capabilities for spatially registering each tool and other controlfeatures (e.g., raise or lower the whole imaging volume with respect tothe position of the radiologist's head display unit). It has multiplebuttons with multiple functions (e.g., easily toggle between controlitem (e.g., total volume; sub-volume; 3D cursor; and, focal pointconvergence) and image settings (e.g., window/leveling; and, filtering,etc.). The master control platform 310 would be equipped with asend/receive element 314 and an inertial measurement unit (IMU) 315. Allother tools are spatially-registered to the master control platform 310,such that they are equipped with registration point(s) 312, asend/receive element 314 and an IMU 315 for position (i.e., translationin the x-direction, y-direction or z-direction) and orientation (i.e.,roll, pitch and yaw) tracking. Next, is the HDU 316 (e.g., augmentedreality, virtual reality, mixed reality) also equipped with registrationpoint(s) 312, send/receive element 314, an IMU 315 and a left eye image318 and a right eye image 320. Next, is the virtual image 322, whichappears as a floating 3D volume in front of the radiologist as a virtualimage on the HDU 316. Next, is the focal point pen 324, which isdirected into the virtual image. This can be used for efficientinteraction with the image, such as selecting objects, guiding focalpoint convergence, write notes, place symbols, etc. As with the othertools, the focal point pen is also equipped with registration point(s)312 and a send/receive element 314 and an IMU 315. Fifth, is thegeo-registered platform 326, which can be used to move a sub-volume inany position or orientation (e.g., place an unknown mass inside of a 3Dcursor and onto the hand-held geo-registered platform, then move theobject to a position that is best suited for close inspection such as 15inches away from the radiologist's eyes, rotate to look at the virtualobject from the top, side, bottom, back, etc.). The geo-registeredplatform is also equipped with registration point(s) 312 andsend/receive element(s) 314 and an IMU 315. Next is the hand-heldmulti-function tool 328, which can be used as any programmedsurgical-type device (e.g., drill, retractor, etc.), which is equippedwith registration point(s) 312 and send/receive element(s) 314 and anIMU. Next, is the hand-held scalpel/knife 330, which is equipped withregistration point(s) 312, send/receive elements 314 and an IMU 315.Next, is the catheter device 332, which would not necessarily have tohave registration point(s) 312 and send/receive element(s) 314 and anIMU 315 for position and orientation tracking, but it could if the usersdemand it so. Note that each item has options for wireless capabilitiesand battery powered. The virtual image 322 is displayed on the HDU 316,but appears as a 3D object sitting right in front of the radiologist onhis desk.

FIG. 4 illustrates a flow diagram for the use of geo-registered tools tooptimize display of medical imaging examinations. In Step A 400, loadvolumetric medical imaging dataset in accordance with checklist andselect from the available tools discussed in this patent, which toolswould be needed to optimize the display of medical imaging examinations.In Step B 402, perform registration and calibration of each tool withthe master control panel by touching multiple registration point(s) oneach geo-registered tool (e.g., head display unit, focal point pen,pedestal/platform, knife, multi-function tool, catheter, master controlplatform, virtual pointer, 3D cursor) with specific registrationpoint(s) on the master control panel for registration. In Step C 404,perform filtering, segmentation and voxel manipulations, e.g. asdescribed in U.S. Ser. No. 15/904,092 and U.S. Ser. No. 16/195,251, bothof which are incorporated by reference. In Step D 406, for every timestep, provide displays in accordance with movement and operation of thecomponents listed in steps above. Step E 408 is to determine if theexamination of this element of this checklist complete. If the answer isno 410, then the next step 412 is to go to step D 406. If the answer isyes 414, then proceed to Step F, which is to review the next set ofmedical images in accordance with the checklist 416. Step G 418 is todetermine if the review is complete. If the answer is no 420, then thenext step 422 is to proceed to step A. If the answer is yes 424, thenthe stop 426.

FIG. 5 illustrates the geo-registration unit, which we also refer to asthe master control platform. The master control platform 500 consists ofthe following: mount (not shown) equipped with position relative to thehead display unit using a geo-registration point(s) 502; platform(s) 504with roll 522, pitch 524 and yaw 526 and translation capability in thex-direction (i.e., side to side) 516, y-direction (i.e.,forward-to-back) 518 and z-direction (i.e., up-down) 520; joystick(s)506 with roll 522, pitch 524 and yaw 526 (RPY) and translationcapability in the x-direction 516, y-direction 518 and z-direction 520;multiple buttons 508 to easily toggle between control item (e.g., totalvolume; sub-volume; 3D cursor; and, focal point convergence toggling)and image settings (e.g., window/leveling; and, filtering, etc.).Joystick 506 functionality includes the following: a) change theorientation of the 3D cursor roll, pitch and yaw; b) zoom the medicalperson viewpoint in toward the 3D cursor and out away from the cursor;c) invoke convergence; d) raise and lower the 3D cursor as to where itis displayed on the headset; e) change the size, shape, and color of the3D cursor; e) invoke filtering, segmentation, sequencing, statistical,and reporting operations; f) invoke pointer and movement controlthereof; g) annotate one or more 3D cursors within the volume ofinterest; h) invoke icon options. Although not mandatory, the deskgeo-registration device 500 would typically be at a fixed location atthe medical person's work station. Another optional component of thegeo-registration unit 500 would be an additional controller 514, whichwould be an ergonomic controller with buttons and joysticks. Thecoordinate system for the medical images volume would be offset aspecified distance from the desk geo-registration device 500. Theregistration points on the focal point pen and the pedestal/platformwould physically touch the registration point(s) 502 on the deskgeo-registration device during the initialization process. Key elementsof the desk geo-registration device include: the geo-registration point502; the transmit/receive unit (aka, the send/receive element) 510;battery element (not shown); and b) the IMU 512.

FIG. 6 illustrates the geo-registered true stereoscopic head displayunit within the geo-registration coordinate system viewing a 3D cursor.The HDU 600 is equipped with an IMU 602, a transmit/receive element 604and a geo-registration point(s) 606. Thus, for a fixed location of a 3Dcursor with respect to the master control unit, a movement of theradiologist's head will alter the appearance of the 3D cursor on the HDU600. The HDU 600 is illustrated in this figure. Key components include:an IMU 602; lenses 610 that display both the real-world scene and thevirtual image 608; geo-registration point 606; battery element (notshown); and digital transmit/receive system 604. The IMU 602 senses headmotion and transmits changes of head position and orientation throughthe transmission system to the master control platform. The computercalculates the effect of changes of head position and orientation andchanges what is being displayed on the lenses and transmits the adjusteddisplay to the HDU 600 to project on the lenses 608. What is beingprojected on the lenses displays is also affected by commands issuedthrough the joystick/master control platform to the computer, and thencean updated display transmitted to the HDU 600. The geo-registrationpoint interacts with the desk geo-registration device and is initializedwith X 616, Y 618, Z 620 coordinates and orientation (i.e., roll 622,pitch 624, and yaw 626) at time of initialization. Note: thesecoordinates and orientation are re-computed when the medical personviewing the medical images puts on the HDU.

FIG. 7 illustrates the geo-registered focal point pen 142 in greaterdetail. The focal point pen 700 is equipped with a geo-registrationpoint 706 at the tip, contains an IMU 702 for determining locationand/or orientation, and a transmit/receive unit 704 for communicationwith the computer. The focal point pen 700 can be moved within the 3Dvolume and point to anomalous tissue 712 and inscribe notes 708 within3D space, but typically adjacent to the volume of interest 710 forfuture reference and to place it into the report. The geo-registrationpoint 706 interacts with the master control platform and is initializedwith X 716, Y 718, Z 720 coordinates and orientation (i.e., roll 722,pitch 724, and yaw 726) at time of initialization. Note: thesecoordinates and orientation are re-computed when the medical personviewing the medical images puts on the HDU. The focal point pen 700which is an actual, tangible object in the shape of a pen (or otheractual object that could be used for pointing) would be held by medicalperson viewing the medical images, which would interact with the virtualmedical images. (Note that the focal point pen 700 is geo-registeredwith the medical images 710.) This interaction includes actuallyphysically moving the focal point pen 700 in the air in front of themedical person viewing the medical images 710 and, simultaneously, bemoving the focal point pen 700 through virtual space showing the 3Dvolumetric medical image 710. The display would show a virtual pen (notshown) properly geo-registered within the 3D medical image. If there ismis-registration between the tangible focal point pen 700 and thevirtual focal point pen (not shown), the focal point pen 700 could bemoved back to the master control platform for re-registration for aprocess including touching the registration points(s) 706 of the focalpoint pen to the registration point of the master control platform (notshown). There is a wide array of uses for the focal point pen whichwould include, but not be limited to, the following: moving the focalpoint pen 700 within the 3D image set so as to follow arteries/veinswithin a complex vascular structure; touching a point within the 3Dimage set with the tip of the focal point pen 700 for annotation and/orcross reference the a particular 2D image slice; writing notes, drawingsymbols (e.g., encircle tissue of concern with a sphere; draw arrows);and, illustrating a potential cut path for surgical planning.

FIG. 8 illustrates multiple calibration points within thegeo-registration coordinate system. In this figure, the focal point pen806 is illustrated touching the location of each of one of thecalibration points 802, which can be inside or outside of the imagingvolume 804. Note that the focal point pen 806 has a registration point808, an IMU 810 and a transmit/receive unit 812.

FIG. 9 illustrates the hand-held pedestal within the geo-registrationcoordinate system. The hand-held pedestal 900 has a geo-registrationpoint 902, an IMU 904 and transmit/receive unit 906, which updates thesystem with regard to its location and orientation. The location of thepedestal can be changed (up/down/left/right/forward/back) and itsorientation (roll, pitch and yaw). This overcomes the difficulty andnon-intuitive interfaces with medical imaging including keyboard, mouse,button on joystick, etc. The radiologist can use a 3D cursor 908 withcopied contents 910, affix it to the pedestal/platform and, transport itto a new location in front of him/her. The geo-registration point 902interacts with the desk geo-registration device and is initialized withX 912, Y 914, Z 916 coordinates and orientation (i.e., roll 918, pitch920, and yaw 922) at time of initialization. Note: these coordinates andorientation are re-computed when the medical person viewing the medicalimages puts on the HDU. The pedestal/platform 900 which is an actualtangible object, such as in the shape of a cell phone (or other actualobject that could be used for holding a virtual object) would be held bymedical person viewing the medical images which would interact with thevirtual medical images. While a geo-registered tool withgeo-registration point(s) 902, an inertial measurement unit 904 andtransmit/receive unit is preferred 906, an alternative embodiment wouldbe to use a set of cameras (e.g., located on the HDU or elsewhere in theroom) for object tracking. (Note that the pedestal/platform 900 isgeo-registered with the medical images.) This interaction includesactually moving pedestal/platform 900 the air in front of the medicalperson viewing the medical images and, simultaneously, be moving thepedestal/platform 900 through virtual space showing the 3D volumetricmedical image. The display would show a virtual pedestal/platform 900properly geo-registered within the 3D medical image. There is a widearray of uses for the pedestal/platform 900 which would include, but notbe limited to, the following: moving the pedestal/platform 900 to avolume of interest within the 3D medical image (e.g., volume 910contained within the 3D cursor 908) by hand movements of thepedestal/platform 900 and then the medical person viewing the medicalimages issues command to affix the volume of interest 910 inside the 3Dcursor 910 to the pedestal/platform 900. Note: once the volume 910 wasaffixed to the pedestal/platform 900, the volume of interest 910 wouldmove, corresponding to and as the pedestal/platform 900 was moved.Thence, the medical person viewing the medical images by hand control ofthe pedestal/platform 900 could rotate, tilt the pedestal/platform 900for examination. Further, this examination process could be accompaniedby head movements by medical person viewing the medical images to obtaina better perspective of any tissue of potential concern. This processallows one to examine a medical imaging dataset the same way that he/shehas spent a lifetime examining hand-held objects, such as a studying thestitches on baseball/softball. The volume on the pedestal/platform 900would return to the original position on the command of the medicalperson. Note: battery in this element is not shown.

FIGS. 10A, 10B, 10C and 10D illustrate an ablative process to aid insearching the internal structure and any abnormalities of an organ. FIG.10A illustrates an organ 1000 contained within the 3D cursor 1002. Toachieve the outer shell of the organ inside the 3D cursor, one canperform a segmentation process to isolate the organ 1000. Then, thesurface layer of voxels can be eliminated, beginning the ablationprocess. The surface layer of voxels can be identified by going fromeither the center voxel of the organ 1004 in the outward direction 1006toward the boundary of the 3D cursor 1002 and analyzing voxel propertiesto determine to voxel at the surface. Alternatively, the surface layerof voxels can be identified by going from the boundary of the 3D cursor1002 in the inward direction 1008 towards the center voxel of the organ1004 and analyzing voxel properties to determine the voxel at thesurface. FIG. 10B shows the organ of interest 1000 without the 3D cursor1002. FIG. 10C sequentially removes voxels from outer shells 1002 of theorgan 1000 in a step-wise fashion. The original outer surface 1010 isshown. Also, the new outer surface 1012 after ablation of N steps isshown. FIG. 10D shows an abnormality 1014 within the confines of theorgan 1000. During each ablative step, normal organ tissue would beablated away, but abnormal liver tissue would remain. In this example,an illustration of a liver lesion called a focal nodular hyperplasia(FNH) 1014 is shown, but all remaining normal liver tissue isdisappeared. For orientation, the original outer surface 1010 is shown.

FIG. 11 illustrates a geo-registered knife and how it could be used tocarve away a portion of a heart. The geo-registered knife 1100 containsa registration point 1102, a transmit/receive element 1104 and an IMU1106. The knife 1100 is a physical object and its position andorientation can be changed by the radiologist. The knife 1100 has theproperties of being able to dissect the virtual images and remove themin order to better view the internal structure of the tissue at hand.For example, the great vessels 1108 could be cut along a cutting plane1110 and rotated away from the remainder of the heart 1112. Thecoordinates of the cutting surface can be determined by the user. Thegeo-registration point interacts with the desk geo-registration deviceand is initialized with X 1114, Y 1116, Z 1118 coordinates andorientation (i.e., roll 1120, pitch 1122, and yaw 1124) at time ofinitialization. Note: these coordinates and orientation are re-computedwhen the medical person viewing the medical images puts on the HDU.Note: battery in this element is not shown.

FIG. 12 illustrates a geo-registered multi-function tool used tomanipulate voxels within the geo-registration coordinate system. Thegeo-registered multi-function tool 1200 is equipped with registrationpoints 1202, an IMU 1204 and a transmit/receive unit 1206. The primaryuse of the geo-registered multi-function tool 1200 is expected to begrabbing tool that can manipulate and hold tissue (i.e., a set ofvoxels) in place. Other surgical instruments, such as drill, hammer,screw, scalpel, etc. can also interface with the tool. As illustrated,two multifunction tools are being used to pull apart two closely spacedblood vessels 1208 with voxel manipulations performed in accordance withU.S. patent application Ser. No. 62/695,868, which is incorporated byreference. The geo-registration point interacts with the deskgeo-registration device and is initialized with X 1210, Y 1212, Z 1214coordinates and orientation (i.e., roll 1216, pitch 1218, and yaw 1220)at time of initialization. Note: these coordinates and orientation arere-computed when the medical person viewing the medical images puts onthe HDU. Note: battery in this element is not shown.

FIG. 13 illustrates the geo-registered catheter with navigation througha blood vessel. The geo-registered catheter 1300 consists of a tubularstructure with a wire entering into it. The geo-registered catheter hasa registration point 1302, an IMU 1304 and a transmit/receive unit 1306.The user's hand 1308 would insert the catheter 1300 into the virtualimage and continuously push it up through the vascular system 1310. Eachsucceeding element of the catheter goes to the location and orientationof the immediately proceeding (or trailing) element as the radiologistpushes, pulls or twists the catheter. Similarly, the virtual catheterwould be able to move through the virtual image via translation in the X1312, Y 1314 or Z 1316 coordinates or via roll 1318, pitch 1320 and yaw1322. This could aid in pre-operative planning or facilitate traininginterventional operations. Note: battery in this element is not shown.

FIG. 14 illustrates geo-registered tool inputs to provide examples oftrue stereoscopic 3D viewing techniques in the “virtual world”. In thecenter of the figure, the HDU 1400 is shown with an initial left eyeview point 1402, left eye viewing angle 1404 and a volume of interest1406 (e.g., volume-subtending 3D cursor) as well as a right eye viewpoint 1408, right eye viewing angle 1410 and the volume of interest 1406(e.g., volume-subtending 3D cursor). Note that the HDU 1400 displays thevolume of interest 1406 with a left eye image 1412 corresponding to theleft eye view point 1402 and left eye viewing angle 1404 of the volumeof interest 1406 and a right eye image 1414 corresponding to the righteye view point 1408 and right eye viewing angle 1410 of the volume ofinterest 1406. In the first example, the orientation of the 3D cursor1406 changes (i.e., change in roll, pitch or yaw) via moving of thehand-held geo-registered platform 1416 temporarily attached to the 3Dcursor 1406. If the radiologist had the image of the heart affixed tothe 3D cursor 1406, then the radiologist could hold the platform 1416and look at the heart from the top, bottom, left and right sides, frontand back. This would provide intuitive controls for viewing an object.Next, the left eye viewpoint 1418 and right eye viewpoint 1420 have beenmoved in toward the 3D cursor 1406, such as the user wearing a HDU 1400with head tracking capabilities physically leaning forward closer to theplatform 1416. Please note that the initial distance between the lefteye viewing point 1402 and right eye viewing point 1408 and the 3Dcursor 1406 has been changed. Alternatively, a similar visual effectcould be achieved via physically moving the platform 1416 and the 3Dcursor 1406 affixed to it closer to the HDU 1400. Thus, movement of theHDU 1400 or platform 1416 could alter the viewing of the 3D cursor 1406.Next, if the radiologist wanted to take a closer inspection through thevolume of interest as defined by the 3D cursor 1408, he could move thefocal point pen 1418 in front of him, such that the tip of the focalpoint pen 1418 is a convergence point 1424. Note that the left eyeviewing angle 1420 and right eye viewing angle 1422 have changed whencompared with the initial left eye viewing angle 1402 and initial righteye viewing angle 1408 in accordance with the focal point convergence,as described in U.S. patent applications Ser. Nos. 12/176,569 and14/313,398 which are incorporated by reference. Next, the radiologistmight decide to raise and lower the 3D cursor as to where it isdisplayed on the headset. This could be accomplished by raising orlowering the radiologist's chair and keeping the radiologist's desk,master control platform, and all other geo-registered tools in a fixedposition. Further, the sub-volume of interest could be separated fromthe rest of the volume, placed on a pedestal and raised or lowered.Further, it could be copied and affixed to the platform 1416, such thatthe original volumetric dataset is unaltered, but the sub-volume iscopied and set aside on the platform for additional inspection/viewing.Further, a virtual pedestal could be invoked through use of the mastercontrol panel to move a sub-volume encompassed in the 3D cursor. Notethat the 3D cursor is illustrated in a lower position in the left eyeimage 1426 and right eye image 1428 in the HDU 1400, compare with theposition of the 3D cursor in the initial left eye image 1412 and initialright eye image 1414. Next, the radiologist could change the size,shape, and color of the 3D cursor, such as using the geo-registeredmulti-function tool 1430 to pinch the 3D cursor 1432 to make it smaller.The master control platform and other geo-registered tools could alsoaccomplish this. Next, the radiologist could invoke filtering,segmentation, sequencing, statistical, and reporting operations 1434,which can be performed via a variety of geo-registered tools. Next, theradiologist could invoke pointer movement control thereof, such asaffixing the virtual pointer 1436 to the focal point pen 1438 so that asthe radiologist moves the geo-registered focal point pen 1438 thathe/she is holding in his/her hand, so does the virtual pointer 1436moves through space in the virtual images 1440 projected on the headdisplay unit. Note that new position of the focal point pen 1444 and thenew position of the virtual pointer. Next, the radiologist could performcutting through moving the geo-registered cutting tool 1450, which islinked to a virtual scalpel 1448 through the volume 1446 to performvirtual dissection. Note that many of these techniques may include voxelmanipulations, which may be implemented as described in U.S. patentapplication Ser. No. 62/695,868 which is incorporated by reference. Thefinal example provided in this figure is for the radiologist to performmovement of a geo-registered catheter 1458 associated with thegeo-registered catheter supporting device 1456, which is linked to avirtual catheter 1452 inside of a virtual blood vessel 1454, which couldbe used to perform dry runs/pre-operative planning sessions.

FIGS. 15A and 15B illustrate digital convergence of the head displayunit to a focal point. Key binocular depth components includestereopsis, convergence and accommodation. This figure illustratesimproving the binocular depth component by employing convergence, suchthat both eyes converge to a single focal within the medical image. Thekey advantages of convergence is that this process replicates how theeyes have been trained over one's lifetime to examine a small area tobetter understand its structure. When examining medical images this isparticularly useful when medical personnel examine small tumors, heartvalves, aneurysms, etc. This figure illustrates a process to replicatethe process of the human eyes when converging to focus on a specificspot but, instead, with 3D medical images. The extra-ocular muscles ofthe eye control the eye movement and direction. In order to provideconvergence to a focal point, some aspects of the left and right eyedisplays could be turned on and other aspects of the left and right eyedisplays could be turned off. In FIG. 15A, assume that an individual islooking straight ahead to a point at an infinite distance. Theorientation of the left eye 1500 and right eye 1502 are parallel. Notethat portions 1504 of the medial side of the left eye display 1501 areturned off and are shown in black, which may account for a small amount(e.g., less than 10 degrees of the horizontal field of view). Note thatportions 1506 of the medial side of the right eye display 1503 areturned off and are shown in black, which may account for a small amount(e.g., less than 10 degrees of the horizontal field of view). Note thatportions 1508 of the lateral side of the left eye display 1501 areturned off and are shown in black, which may account for a small amount(e.g., less than 10 degrees of the horizontal field of view). Note thatportions 1510 of the lateral side of the right eye display 1503 areturned off and are shown in black, which may account for a small amount(e.g., less than 10 degrees of the horizontal field of view). Note thatportions 1512 of the central aspect of the left eye display 1502 areturned on and are shown in white, which may account for a large amount(e.g., 90 degrees of the horizontal field of view). Note that portions1514 of the central aspect of the right eye display 1503 are turned onand are shown in white, which may account for a large amount (e.g., 90degrees of the horizontal field of view). In FIG. 15B, a black dot 1516is the focal point and is shown in both the left eye display 1518 andright eye display 1520. The orientation of the left eye 1522 and righteye 1524 are both angled inward, illustrating the process of theextra-ocular muscles moving the eyes inward when viewing an object in 3Dspace that is close by. Note that portions 1526 of the lateral side ofthe left eye display 1518 are turned off and are shown in black, whichmay account for a small amount (e.g., less than 10 degrees of thehorizontal field of view). Note that portions 1528 of the lateral sideof the right eye display 1520 are turned off and are shown in black,which may account for a small amount (e.g., less than 20 degrees of thehorizontal field of view). Note that portions 1530 of the central andmedial aspect of the left eye display 1518 are turned on and are shownin white, which may account for a large amount (e.g., 90 degrees of thehorizontal field of view). Note that portions 1532 of the central aspectof the right eye display 1520 are turned on and are shown in white,which may account for a large amount (e.g., 90 degrees of the horizontalfield of view). Thus, the portions of the field of view utilized can bealtered in accordance with convergence. Thus, an example (but notlimited to) of the convergence process is shown in this figure. First,from the left eye perspective: a) relative to looking straight ahead,occluding/eliminating the left portion of the display and associatedvoxels which would have been displayed with additional voxels displayedon the right portion of the display. Similarly, for the right eyeoccluding/eliminating the right portion of the display and addingadditional voxels to the left portion. b) shifting the display of theconvergence/focal point to the right for the left eye of the centerpoint of the display proportional to the angular change based ondistance of the view point from the convergence/focal point, similarlyshifting the convergence/focal point to the left for the right eye; c)reducing the total voxels displayed for both eyes to reflect changes inthe field of view when observing a close object.

FIGS. 16A and 16B illustrate a process of providing increased resolutionin the field of view corresponding to the fovea and is linked to thefocal point. The human eye fovea is a small spot on the back of the eyewith high visual acuity with an angular field of view of approximately 5degrees. Away from the fovea, the visual acuity is not as sharp. Thus, adisplay which better utilizes this feature of the human fovea would havevalue. The visual acuity decreases at farther distances from the foveain a non-linear fashion. Multiple types of HDUs could be utilized. Inthis example, the term pixels is used, but other types of HDUs could usethis property. Specifically, the degrees of angular resolution in thefield of view per pixel could likewise be variable. An example will beillustrated. Assume that there are 1000 pixels across the display in ahorizontal fashion. Assume that the horizontal field of view is 80degrees. If 250 pixels were allocated to the 5 degree high resolutionband (based on the approximate field of view of the fovea), then eachpixel in this region would subtend 0.02 degrees. The remaining 750pixels would be allocated to the 75 degrees of lower resolution band,such that each pixel in this region would subtend 0.1 degrees. As thefocal point pen moves to different regions in the display, thehigh-resolution band would change accordingly. In FIG. 16A, the left eye1600 and right eye 1602 are oriented in a parallel fashion, which wouldbe done when looking at an object at an infinite distance, such as thehorizon. Note that portions 1604 of the medial side of the left eyedisplay 1601 are turned off and are shown in black, which may accountfor a small amount (e.g., 5 degrees of the horizontal field of view).Note that portions 1606 of the medial side of the right eye display 1603are turned off and are shown in black, which may account for a smallamount (e.g., 5 degrees of the horizontal field of view). Note thatportions 1608 of the lateral side of the left eye display 1601 areturned off and are shown in black, which may account for a small amount(e.g., 5 degrees of the horizontal field of view). Note that portions1610 of the lateral side of the right eye display 1603 are turned offand are shown in black, which may account for a small amount (e.g., 5degrees of the horizontal field of view). Note that a black dot is shown1614 in the left eye display 1601, which corresponds to the horizontalcenter and vertical center of the left eye field of view. Note that ablack dot is shown 1616 in the right eye display 1601, which correspondsto the horizontal center and vertical center of the right eye field ofview. Note that a gray square 1618 shown within the left eye display1601 corresponds to the high-resolution portion of the FOV (e.g., 0.02degrees per pixel). The remainder of the left eye display 1601, which isturned on 1612 would have a lower resolution portion of the FOV (e.g.,0.1 degrees per pixel). Note that a gray square 1620 shown within theright eye display 1603 corresponds to the high-resolution portion of theFOV (e.g., 0.02 degrees per pixel). The remainder of the right eyedisplay 1603, which is turned on 1614 would have a lower resolutionportion of the FOV (e.g., 0.1 degrees per pixel). Thus, FIG. 16Aillustrates Pixel display for one side of the HDU with a FOV of 80° andenhanced FOV of 5° given a disproportionate high number ofpixels/angular FOV shown as the gray rectangle 1618 in the left eyedisplay 1601 and the gray rectangle 1620 in the right eye display 1603.FIG. 16B illustrates the left eye 1622 and right eye 1624 both lookingdownward and to the right side. Note that portions 1626 of the lateralside of the left eye display 1605 are turned off and are shown in black,which may account for a small amount (e.g., 10 degrees of the horizontalfield of view). Note that portions 1628 of the medial side of the righteye display 1607 are turned off and are shown in black, which mayaccount for a small amount (e.g., 10 degrees of the horizontal field ofview). Note that a black dot is shown in the left eye display 1605,which corresponds to the convergence point 1630. Note that a black dotis shown in the right eye display 1607, which corresponds to theconvergence point 1630. Note that a gray square 1632 shown within theleft eye display 1605 corresponds to the high-resolution portion of theFOV (e.g.,0.02 degrees per pixel). The remainder of the left eye display1605, the portion which is turned on 1634, would have a lower resolutionportion of the FOV (e.g., 0.1 degrees per pixel). Note that a graysquare 1636 shown within the right eye display 1607 corresponds to thehigh-resolution portion of the FOV (e.g., 0.02 degrees per pixel). Theremainder of the right eye display 1607, which is turned on 1638 wouldhave a lower resolution portion of the FOV (e.g., 0.1 degrees perpixel). Thus, FIG. 16B illustrates a display wherein the angular fieldof view per area is variable. Thus, some regions in the display will beallocated higher number of pixels/angular FOV and some regions in thedisplay will be allocated a lower number of pixels/angular FOV. Anexample wherein this may prove beneficial is during inspection of aclose structure, such as a carotid artery for atherosclerotic plaques.The tip of the geo-registered focal point pen could be used to help theeyes track along a structure (e.g., carotid artery) to help the eyesfollow along the structure and inspect for abnormalities. Thegeo-registered focal point pen would control the high resolution FOV andenhance overall detection.

FIGS. 17A and 17B illustrate a visual representation of convergence. InFIG. 17A, the volume of interest 1700 is shown. Note that left eye viewpoint 1702 and right eye view point 1704 are shown. Note the left eyeviewing angle 1706 and right eye viewing angle 1708 are shown. Note theconvergence point 1712. Note that a center line is shown extending froma point on the plane between the eyes 1714 to (or near) the convergencepoint 1712. This line may help focus the user's attention. In FIG. 17B,the center line of one user could be displayed on all user's HDUs in amulti-HDU user situation. This would enable one user to see anotheruser's center line. This could facilitate communication between multipleusers. The center line 1710 would be placed in a fashion that would aidthe user in their attention and their focus. For example, a center line1720 appearing from overhead towards the object of interest (e.g., nearthe focal point 1718) may be the optimal placement. Note that the centerline 1720 would only be visible to those wearing HDUs 1716 and wouldappear as a 3D structure in the left eye display 1722 and right eyedisplay 1724.

FIGS. 18A, 18B, 18C, and 18D illustrate methods of helping theradiologist in search pattern via utilization of saccades searchtechnique. The example shown is a branching carotid artery arterialstructure wherein the common carotid artery 1800, carotid bulb 1802,internal carotid artery 1804 and external carotid artery 1806 are shown.In FIG. 18A, a first black dot 1808 would appear at a first time point.The appearance of a first black dot 1808 (or other similar type object)would draw the eye in towards the new structure. This would force thehuman eye (and fovea region) to look in the region of the first blackdot 1808 and examine those local structures, namely the common carotidartery 1800. In FIG. 18B, the first black dot 1808 disappears and asecond black dot 1810 would appear at a second time point. Theappearance of a second black dot 1810 (or other similar type object)would draw the eye in towards the new structure. This would force thehuman eye (and fovea region) to look in the carotid bulb 1802. In FIG.18C, the second black dot 1810 disappears and a third black dot 1812would appear at a third time point. The appearance of a third black dot1812 (or other similar type object) would draw the eye in towards thenew structure. This would force the human eye (and fovea region) to lookin the internal carotid artery 1804. In FIG. 18D, the third black dot1812 disappears and a fourth black dot 1814 would appear at a fourthtime point. The appearance of a fourth black dot 1814 (or other similartype object) would draw the eye in towards the new structure. This wouldforce the human eye (and fovea region) to look in the region of theexternal carotid artery 1806. Thus, utilization of planned structuresthat pop up on an image at strategic points would therefore use thehuman eye's natural ability to perform saccades and utilize the fovea.Segmentation algorithms could be utilized, and dots strategicallypositioned at sites where pathology is detected (e.g., by an AIalgorithm) or where pathology is statistically most likely to occur(e.g., atherosclerosis in the carotid bulbs). Furthermore, the methodthat the radiologist could implement to help with the saccades includeson a set time (e.g., a new dot appears every 2 seconds) or could be byuser control (e.g., user clicks a mouse and a new dot appears).Furthermore, the dots could be tied to a radiologist's checklist, suchthat when all dots are examined for a particular structure, evaluationof that structure would be complete. Furthermore, an eye tracking systemcould be utilized to help determine the optimum tools for lesiondetection (e.g., whether it be saccades or smooth tracking, see FIG. 19,or combination thereof).

FIGS. 19A, 19B, 19C, and 19D illustrate a method of helping theradiologist in search pattern via utilization of smooth tracking searchtechnique. The example shown is an arterial structure wherein the commoncarotid artery 1900, carotid bulb 1902, internal carotid artery 1904 andexternal carotid artery 1906 are shown. In FIG. 19A, a line 1908 isshown coursing from the common carotid artery 1900 through the carotidbulb 1902 and into the internal carotid artery 1904. The line 1908 wouldbe an optional feature and would not be required for smooth tracking,and could be displayed or hidden by user preference. A black dot 1910(or similar visual structure) is shown at the proximal portion of thecommon carotid artery 1900 at an initial time point. In FIG. 19B, theblack dot 1910 is shown to be moving along that line and is now at thelevel of the carotid bulb 1902. Note that the black dot 1910 would beshown continuously and moved in a continuous fashion with a frame ratefast enough that the human eye sees smooth movement. In FIG. 19C, theblack dot 1910 is shown to be moving along that line and is now at thelevel of the internal carotid artery 1904. Note that the black dot 1910would be shown continuously and moved in a continuous fashion with aframe rate fast enough that the human eye sees smooth movement. Afterscanning the course of the common carotid artery 1900, carotid bulb 1902and internal carotid artery 1904 for abnormalities, the radiologist maythen elect to scan the external carotid artery 1906. A new line 1912 anda new black dot 1914 would then be used for scanning of the nextstructure. This new line 1912 and new black dot 1914 would suddenlyappear at the new location and the human eye would perform a saccadesmovement to the new items. Then, the new black dot 1914 would smoothlymove along the course of the external carotid artery 1906 in acontinuous, smooth fashion with a frame rate fast enough that the humaneye sees smooth movement. This would be analogous to a patientperforming smooth tracking of a doctor's finger. Thus, a combination ofsaccades and smooth tracking eye movements can be utilized to help theradiologist improve visual tracking of abnormalities within structures.The rate of the smooth tracking and movement of the black dot would becontrolled by the radiologist via adjusting input settings.Alternatively, this type of tracking could be linked to the movement ofa focal point pen within the image. The human can move the black dot(via the focal point pen or GUI) or the computer or can control theblack dot to aid the human in performing smooth tracking and assessmentof the structure. Also, the radiologist can tab through various pointsof interest within the sub-volume as desired. This act will serve tomimic the human eyes' natural movement of performing saccades from oneitem of interest to another item of interest.

FIGS. 20A and 20B illustrate the capability of using the geo-registeredplatform to display a known pathology from a database next to anothergeo-registered platform with an unknown pathology from the patient'sscan. In this figure, a first 3D cursor 2000 is shown affixed to a firstgeo-registered platform 2002. The first 3D cursor 2000 contains anunknown lesion 2004. For example, the unknown lesion 2004 could be abreast mass, but the precise diagnosis of the breast mass is not known.A second 3D cursor 2006 is shown affixed to a second geo-registeredplatform 2008. The second 3D cursor 2006 contains a known lesion 2010.For example, the known lesion 2010 could be a breast mass and theprecise diagnosis of the breast mass is known to be an infiltratingductal carcinoma. Note that the margins of the known mass 2010 in thisexample are spiculated whereas the margins of the unknown mass 2004 arelobulated. The radiologist would conclude from this comparison that thepatient's pathology in the unknown lesion 2004 is different from thepathology in the known lesion 2010. Thus, the radiologist would have theability to place the known pathology lesion 2010 on one pedestal. Thiscan be imported from a known pathology database. The radiologist wouldhave the ability to place unknown pathology lesion 2004 on anotherpedestal. This is from the patient's scan. These two could be comparedin a side-by-side fashion.

Several features, aspects, embodiments, and implementations have beendescribed. Nevertheless, it will be understood that a wide variety ofmodifications and combinations may be made without departing from thescope of the inventive concepts described herein. Accordingly, thosemodifications and combinations are within the scope of the followingclaims.

What is claimed is:
 1. A method comprising: determining a convergencepoint corresponding to a user's fovea; generating an image wherein afirst portion of the image has a first pixel resolution and a secondportion of the image has a second pixel resolution wherein the firstpixel resolution is higher than the second pixel resolution; presentingthe image on a display; and aligning the first portion of the image withthe convergence point corresponding to the user's fovea.
 2. The methodof claim 1 further comprising wherein: during a first time epoch, thefirst portion of the image has a first location within the image; andduring a second time epoch, the first portion of the image has a secondlocation within the image.
 3. The method of claim 1 further comprisingwherein: during a first time epoch, the first portion of the imagesubtends a first angular field of view within the image; and during asecond time epoch, the first portion of the image subtends a secondangular field of view within the image.
 4. The method of claim 1 furthercomprising wherein the first portion of the image subtends an angularfield of view of approximately 5 degrees.
 5. The method of claim 1further comprising a non-linear transition between the first pixelresolution in the first portion of the image and the second pixelresolution of the second portion of the image.
 6. The method of claim 1further comprising wherein the location of the first portion of theimage is determined via eye tracking.
 7. The method of claim 1 furthercomprising wherein the location of the first portion of the image isdetermined by the user's look angle.
 8. The method of claim 1 furthercomprising wherein the location of the first portion of the image isdetermined by the convergence point.
 9. The method of claim 1 furthercomprising wherein the location of the first portion of the image isdetermined by a digital object's position within the image.
 10. Themethod of claim 1 further comprising wherein the location of the firstportion of the image is determined by a geo-registered focal point pen'sposition.
 11. The method of claim 1 further comprising wherein thedisplay is an extended reality head display unit.
 12. The method ofclaim 1 further comprising wherein the display contains a section notdisplayed with pixels.
 13. The method of claim 12 further comprisingwherein the section not displayed with pixels is determined by theuser's look angle.
 14. The method of claim 12 further comprising whereinthe section not displayed with pixels is determined by the convergencepoint.
 15. The method of claim 12 further comprising wherein the sectionnot displayed with pixels is determined by a digital object's positionwithin the image.
 16. The method of claim 12 further comprising whereinthe section not displayed with pixels is determined by a geo-registeredfocal point pen's position.
 17. A non-transitory computer readablemedium having computer readable code thereon for generating an image,the medium comprising: instructions for determining a convergence pointcorresponding to a user's fovea; instructions for generating an imagewherein a first portion of the image has a first pixel resolution and asecond portion of the image has a second pixel resolution wherein thefirst pixel resolution is higher than the second pixel resolution;instructions for presenting the image on a display; and instructions foraligning the first portion of the image with the convergence pointcorresponding to the user's fovea.
 18. The non-transitory computerreadable medium of claim 17 further comprising wherein: during a firsttime epoch, the first portion of the image has a first location withinthe image; and during a second time epoch, the first portion of theimage has a second location within the image.
 19. A head display unit(HDU) comprising: a processor; a left eye display; a right eye display;a non-transitory memory having computer-executable instructions storedthereupon which, when executed by the processor of the HDU, cause theHDU to: generate a first image wherein a first portion of the firstimage has a first pixel resolution and a second portion of the firstimage has a second pixel resolution wherein the first pixel resolutionis higher than the second pixel resolution; generate a second imagewherein a first portion of the second image has a third pixel resolutionand a second portion of the second image has a fourth pixel resolutionwherein the third pixel resolution is higher than the fourth pixelresolution; and cause the first image to be displayed on the left eyedisplay and cause the second image to be displayed on the right eyedisplay, wherein the first image is aligned with the left eye of a user,wherein the first portion of the first image is aligned with the foveaof the left eye of the user, wherein the second image is aligned withthe right eye of the user, wherein the first portion of the second imageis aligned with the fovea of the right eye of the user.
 20. The HDU ofclaim 19 further comprising wherein: during a first time epoch, thefirst portion of the first image has a first location within the firstimage and the first portion of the second image has a first locationwithin the second image; and during a second time epoch, the firstportion of the first image has a second location within the first imageand the first portion of the second image has a second location withinthe second image.